Predicting probability of occurrence of a string using sequence of vectors

ABSTRACT

Systems and methods are disclosed to predict probability of occurrence of a string. A sequence of vectors is generated based at least on a maximum length of word for each symbol in the string. The sequence of vectors is provided to a machine learning unit for the string. A probability of occurrence of the string is obtained from the machine learning unit.

RELATED APPLICATIONS

The present application is a Continuation of U.S. patent applicationSer. No. 16/021,689, filed Jun. 28, 2018 and claims priority to RussianPatent Application No.: RU2018123353, filed Jun. 27, 2018, the entirecontents of which are hereby incorporated by reference herein.

TECHNICAL FIELD

The present disclosure is generally related to computer systems, and ismore specifically related to systems and methods for predictingprobability of occurrence of a string using a language model.

BACKGROUND

A language model may be used to predict the probability distribution ofa given linguistic units, such as, symbols, words, sentences, etc. Forexample, a probabilistic language model can predict the next word in asequence of words given the words that precede it. A language model canassign the probability for the likelihood of a given word or symbol tofollow a sequence of words or symbols. A language model can learn theprobability based on examples of text, speech, etc.

SUMMARY OF THE DISCLOSURE

In accordance with one or more aspects of the present disclosure, anexample method may comprise: receiving a plurality of strings, eachstring of the plurality of strings comprising a plurality of symbols;for each string of the plurality of strings, generating, by a processingdevice, a first sequence of vectors based at least on a maximum lengthof word for each symbol in the string; providing to a machine learningunit the first sequence of vectors for each string of the plurality ofstrings; and obtaining from the machine learning unit a probability ofoccurrence of each string of the plurality of strings.

In accordance with one or more aspects of the present disclosure, anexample system may comprise: a memory device storing instructions; aprocessing device coupled to the memory device, the processing device toexecute the instructions to: receive a plurality of strings, each stringof the plurality of strings comprising a plurality of symbols; for eachstring of the plurality of strings, generate a first sequence of vectorsbased at least on a maximum length of word for each symbol in thestring; provide to a machine learning unit the first sequence of vectorsfor each string of the plurality of strings; and obtain from the machinelearning unit a probability of occurrence of each string of theplurality of strings.

In accordance with one or more aspects of the present disclosure, anexample non-transitory computer-readable storage medium may compriseinstructions that, when executed by a processing device, cause theprocessing device to: receive a plurality of strings, each string of theplurality of strings comprising a plurality of symbols; for each stringof the plurality of strings, generate a first sequence of vectors basedat least on a maximum length of word for each symbol in the string;provide to a machine learning unit the first sequence of vectors foreach string of the plurality of strings; and obtain from the machinelearning unit a probability of occurrence of each string of theplurality of strings.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated by way of example, and not by wayof limitation, and can be more fully understood with reference to thefollowing detailed description when considered in connection with thefigures in which:

FIG. 1 depicts a high-level component diagram of an illustrative systemarchitecture, in accordance with one or more aspects of the presentdisclosure.

FIG. 2 depicts a flow diagram of one illustrative example of a methodfor predicting probability of occurrence of a string, in accordance withone or more aspects of the present disclosure.

FIG. 3 depicts one illustrative example of receiving a plurality ofstrings and obtaining a probability of occurrence for each of theplurality of strings, in accordance with one or more aspects of thepresent disclosure.

FIG. 4 depicts one illustrative example of generating a sequence ofvectors to provide to a machine learning unit to obtain a probability ofoccurrence, in accordance with one or more aspects of the presentdisclosure.

FIG. 5 depicts an example computer system which can perform any one ormore of the methods described herein, in accordance with one or moreaspects of the present disclosure.

DETAILED DESCRIPTION

A language model may be used to predict the probability distribution ofa given linguistic units, such as, symbols, words, sentences, etc.Language modeling is used for various natural language processing tasks,such as, machine translation, speech recognition, optical characterrecognition, spelling check, handwriting recognition, etc. The use ofneural networks in language modeling may be known as neural languagemodeling or neural net language modeling. Neural language models may usecontinuous representations or word embeddings to predict the probabilityof occurrence of a particular language unit, such as, a symbol, word,sentence, etc. Word embeddings are used to represent words or phrasesfrom natural language vocabulary by mapping the words or phrases tovectors of real numbers. Neural network language models may beconstructed and trained as probabilistic classifiers that learn topredict a probability distribution. A probabilistic classifier may be amachine learning classifier that can predict a probability distributionover a set of classes given an observation of an input, rather than onlyoutputting the most likely class that the observation should belong to.

Conventionally, a neural network used in a neural language model can betrained to predict a probability distribution over the linguistic unitgiven some linguistic context. A linguistic context may be defined interms of what follows or what precedes a particular linguistic unit. Forexample, a context may be a fixed-sized window of previous words. Inanother example, a context may be a fixed-size window of future words.For example, the fixed-size window may consist of four words. In somecases, each word may contain a specified number of letters, such as fiveletters, for example. In an example, the fixed-size window may consistof one word. In that case, the context may be the word that immediatelyprecedes or follows the target word or symbol. Traditionally, languagemodeling may involve analyzing a particular linguistic unit (e.g., thetarget word or symbol) by taking into account a context that precedes aword or symbol, and/or the context that follows a word or symbol topredict the probability of occurrence of the particular linguistic unitor a portion of it. In an example using both previous and futurecontext, the probability value may be expressed as:P(U_(t)|C_(P),C_(f)),

-   -   where P is the probability of occurrence of U_(t) given the        context C_(P) and C_(f),    -   U_(t) is the target unit (e.g., symbol, string, word, etc.),    -   C_(P) is the preceding context,    -   C_(f) is the following context.

The systems and methods described herein represent significantimprovements in better predicting probability of occurrence of alinguistic unit (e.g., a string, a word) by not only using the previousand/or future context, but also using maximum lengths of a word thatbegins with each symbol in the linguistic unit (e.g., a string, a word).The mechanisms provided herein can train a machine learning model (e.g.,a neural network) to predict, given a plurality of strings, theprobability of occurrence of each string of the plurality of strings.The mechanism may include receiving a plurality of strings for whichprobabilities of occurrence values are to be predicted. The plurality ofstrings may each contain plurality of symbols. The symbols may followone another to form each string. The plurality of strings may vary fromeach other by one symbol. The one symbol may be located on the sameposition of each string. The strings may be sent to a machine learningunit to predict the probability of occurrence of each string. Prior tosending the strings to the machine learning unit, each string may berepresented as a sequence of vectors.

In some implementations, the mechanisms may generate a sequence ofvectors comprising one or more vectors. Each vector in the sequence ofvectors may correspond to each symbol in a given string. Each vector inthe sequence of vectors may be derived by joining together a firstvector and a second vector for a given symbol in the string. The firstvector may comprise a maximum length of word for a given symbol in agiven string. The maximum length may correspond to the length of thelongest possible word (e.g., a word that can be found in a dictionary)within the string that begins with the given symbol. The second vectormay comprise a symbol vector for the given symbol in the given string.The symbol vector may be formed based on one or more of: symbolembeddings, unified vector for symbols of an alphabet different from thealphabet used in the string, vectors for frequently occurringpunctuation marks, a vector for rare punctuation marks, vectors fornumerals, etc. The first and second vectors for each symbol may becombined together into one vector for each symbol.

After generating the sequence of vectors, the mechanisms may provide thesequence of vectors to a machine learning unit as an input. In someimplementations, the machine learning unit may comprise fully connectedneural networks with multiple layers. The neural network may receiveeach sequence of vectors for each string and apply sequence of matrixtransformations. In an implementation, the machine learning unit maycomprise two fully connected layers and a fully connected output layer.Batch normalization and rectifier linear unit (ReLU) functions may beapplied to the outputs of the initial two fully connected layers and asigmoid activation function may be applied on the output of the outputlayer. The machine learning unit may provide as an output a probabilityof occurrence of each string of the plurality of strings. Theprobability value may be expressed using a vector, where each vectorcomponent corresponds to one string. The value of the component of thevector that is closest to the value ‘one’ compared to the values of theother components corresponds to the string having the highestprobability of occurrence. Thus, if the plurality of strings reflectsdifferent possibilities of a word to be recognized, the string thatproduces the highest probability value may be the string that representsthe most likely possibility of word recognition.

The terms “character,” “symbol,” “letter,” and “cluster” may be usedinterchangeably herein. A cluster may refer to an elementary indivisiblegraphic element (e.g., graphemes and ligatures), which are united by acommon logical value. Further, the term “word” may refer to a sequenceof symbols, and the term “sentence” may refer to a sequence of words.

The neural language model may be used in various stages of naturallanguage processing (NLP), such as post-processing of the recognizedtext, spell-checking, text recognition, text generation, translation,etc. The method is easily applied to multiple use cases. The techniquesdescribed herein allow for improved accuracy of prediction ofprobabilities of occurrence in language modeling. Using the maximumlength of words produces improved predictions of probabilities becauseof the use of words with commonplace meaning, such as, words found in adictionary. A dictionary contains a comprehensive repository of wordswith actual meanings, while a neural network with relatively smalldataset may not be trained to remember each of those words. Thus, crossreferencing the words with dictionary words and using the maximum lengthof the words found in a dictionary provide for a simple networktopology, and the network can be quickly trained on a relatively smalldataset.

Various aspects of the above referenced methods and systems aredescribed in details herein below by way of examples, rather than by wayof limitation.

FIG. 1 depicts a high-level component diagram of an illustrative systemarchitecture 100, in accordance with one or more aspects of the presentdisclosure. System architecture 100 includes a computing device 120, arepository 160, and a server machine 150 connected to a network 130.Network 130 may be a public network (e.g., the Internet), a privatenetwork (e.g., a local area network (LAN) or wide area network (WAN)),or a combination thereof.

The computing device 120 may perform prediction of probability ofoccurrence using artificial intelligence. In one embodiment, computingdevice 120 may be a desktop computer, a laptop computer, a smartphone, atablet computer, a server, a scanner, or any suitable computing devicecapable of performing the techniques described herein. Computing device120 may receive a plurality of strings. In an example, strings 110 maybe received by the computing device 120. In some implementations,strings 110 may be produced by an alpha-numeric input device, a voicedevice, or another device capable of producing a plurality of strings.Each string in strings 110 may comprise a plurality of symbols. In anexample, a particular word on an image of a document may need to berecognized. A text recognition system may produce a set of strings aspotential candidates for the word to be recognized. To make an ultimatedecision on which one of these potential candidates should be selectedas a result of word recognition, the set of strings may be provided asstrings 110 as an input to computing device 120. Computing device 120may predict a probability of occurrence of each of the strings 110 andproduce the probability as an output. The string with the highestprobability of occurrence out of the strings 110 may be selected as theresult of word recognition from the set of potential candidate strings.

In one embodiment, computing device 120 may include a sequence engine122 and a probability engine 124. The sequence engine 122 and theprobability engine 124 may each include instructions stored on one ormore tangible, machine-readable storage media of the computing device120 and executable by one or more processing devices of the computingdevice 120. In one embodiment, sequence engine 122 may generate asequence of vectors based on strings 110. For example, each vector inthe sequence of vectors may be derived by combining a symbol vector fora given symbol in a given string of strings 110 and a vector of maximumlength of word for a given symbol in a given string of strings 110.

In one embodiment, probability engine 124 may use a trained machinelearning model 140 that is trained and used to predict probability ofoccurrence of each of strings 110 given the sequence of vectors forstrings 110 generated by sequence engine 122. The machine learning model140 may be trained using training set of strings and correspondingsequence vectors. In some instances, the machine learning model 140 maybe part of the probability engine 124 or may be accessed on anothermachine (e.g., server machine 150) by the probability engine 124. Basedon the output of the trained machine learning model 140, the probabilityengine 124 may predict the probability of occurrence of each string ofstrings 110.

Server machine 150 may be a rackmount server, a router computer, apersonal computer, a portable digital assistant, a mobile phone, alaptop computer, a tablet computer, a camera, a video camera, a netbook,a desktop computer, a media center, or any combination of the above. Theserver machine 150 may include a training engine 151. The machinelearning model 140 may refer to model artifacts that are created by thetraining engine 151 using the training data that includes traininginputs and corresponding target outputs (correct answers for respectivetraining inputs). During training, patterns in the training data thatmap the training input to the target output (the answer to be predicted)can be found, and are subsequently used by the machine learning model140 for future predictions. As described in more detail below, themachine learning model 140 may be composed of, e.g., a single level oflinear or non-linear operations (e.g., a support vector machine [SVM])or may be a deep network, i.e., a machine learning model that iscomposed of multiple levels of non-linear operations). Examples of deepnetworks are neural networks including convolutional neural networks,recurrent neural networks with one or more hidden layers, and fullyconnected neural networks.

As noted above, the machine learning model 140 may be trained todetermine the probability of occurrence for plurality of strings usingtraining data, as further described below. Once the machine learningmodel 140 is trained, the machine learning model 140 can be provided toprobability engine 124 for analysis of strings 110. For example, theprobability engine 124 may input the sequence of vectors for strings 110being analyzed into the machine learning model 140. In some examples,model 140 may consist of a fully connected neural network with multiplelayers. For example, the neural network may comprise of two fullyconnected layers of the network and a fully connected output layer ofthe network. The probability engine 124 may obtain one or more outputsfrom the trained machine learning model 140. The output may be asequence of probability values indicating probabilities of occurrencesof each string of the strings 110.

The repository 160 may be a persistent storage that is capable ofstoring strings 110 as well as data structures to tag, organize, andindex the strings 110. Repository 160 may be hosted by one or morestorage devices, such as main memory, magnetic or optical storage baseddisks, tapes or hard drives, NAS, SAN, and so forth. Although depictedas separate from the computing device 120, in an implementation, therepository 160 may be part of the computing device 120. In someimplementations, repository 160 may be a network-attached file server,while in other embodiments, repository 160 may be some other type ofpersistent storage such as an object-oriented database, a relationaldatabase, and so forth, that may be hosted by a server machine or one ormore different machines coupled to the via the network 130.

FIG. 2 depicts a flow diagram of one illustrative example of a methodfor predicting probability of occurrence of a string, in accordance withone or more aspects of the present disclosure. Method 200 and/or each ofits individual functions, routines, subroutines, or operations may beperformed by one or more processors of the computer system (e.g.,example computer system 500 of FIG. 5) executing the method. In certainimplementations, method 200 may be performed by a single processingthread. Alternatively, method 200 may be performed by two or moreprocessing threads, each thread executing one or more individualfunctions, routines, subroutines, or operations of the method. In anillustrative example, the processing threads implementing method 200 maybe synchronized (e.g., using semaphores, critical sections, and/or otherthread synchronization mechanisms). Alternatively, the processingthreads implementing method 200 may be executed asynchronously withrespect to each other. Therefore, while FIG. 2 and the associateddescription lists the operations of method 200 in certain order, variousimplementations of the method may perform at least some of the describedoperations in parallel and/or in arbitrary selected orders. In oneimplementation, the method 200 may be performed by one or more of thevarious components of FIG. 1, such as, sequence engine 122, probabilityengine 124, etc.

At block 210, the computer system implementing the method may receive aplurality of strings. Each string of the plurality of strings mayinclude a plurality of symbols. For example, the received plurality ofstrings may be comparable to strings 110 of FIG. 1. Probabilities ofoccurrence values are to be predicted for each string of the pluralityof strings. In some implementations, each string of the plurality ofstrings may differ from each other string of the plurality of strings byone symbol, which may be referred herein as the “target symbol.” Thetarget symbol may be located at the same position of each string.

In some examples, the plurality of strings may correspond to a set ofpotential strings produced by a translation system for a word to betranslated from one language to another. In some examples, the pluralityof strings may correspond to a set of candidate strings produced by atext recognition system for a word to be recognized in an image ofdocument. In an example, a word on an image of document may need to berecognized, for which a text recognition system may produce a set ofcandidate strings consisting of the strings 1) “steamboat,” 2)“s1eamboat,” 3) “sleamboat,” and 4) “sieamboat.” The set of candidatestrings may be received as a plurality of strings from the textrecognition system so that a probability of occurrence of each of thestrings of the set of strings may be predicted. The text recognitionsystem may select the string for which the probability is predicted tobe the highest amongst the set of candidate strings.

For example, FIG. 3 depicts an example of receiving a plurality ofstrings to obtain a probability of occurrence for each of the pluralityof strings, in accordance with one or more aspects of the presentdisclosure. In the example, the computer system 300 implementing amethod to predict probability of occurrences may receive a plurality ofstrings 310. Plurality of strings 310 may include strings 311, 312, 313,and 314. Each string of the plurality of strings 310 may include aplurality of symbols. For example, string 311 may include plurality ofsymbols 311 a-311 i. Each of the plurality of symbols 311 a-311 ifollows one another to form a sequence representing string 311. Eachstring 311, 312, 313, and 314 may differ from each other by one symbol,which may be referred herein as the “target symbol.” The target symbolmay be located at the same position of each string. As shown, for theplurality of strings 310, the target symbol is located at the fifthplace of the sequence of symbols representing each string. The targetsymbol 311 e of string 312 differs from target symbols of the otherstrings 312, 313, and 314 located on the same, fifth position of eachstring. As it is seen, the remaining symbols of each string are same asevery other string at the given location of the symbol.

In one example, each string 311, 312, 313 and 314 may represent acandidate string for a word to be recognized. System 300 may receive thestrings 310 to predict the probability of occurrence of each string311-314 to determine the string corresponding to the highestprobability. In one example, the string corresponding to the highestprobability may be selected as the final recognized word. In anotherexample, some other criteria may be used as it related to the predictedprobability values to determine the selected recognized word.

Referring back to FIG. 2, at block 220, the computer system maygenerate, for each string of the plurality of strings, a first sequenceof vectors based at least on a maximum length of word for each symbol inthe string. Each vector of the first sequence of vectors may correspondto each symbol in the string. Each vector of the first sequence ofvectors may be derived by joining together a first vector comprising themaximum length of word for a given symbol in the string and a secondvector comprising a symbol vector for the given symbol in the string.The maximum length of word for each symbol in the string may correspondto length of a longest possible word within the string that starts withthe symbol. In some examples, the longest possible word is found in adictionary. The symbol vector for each symbol in the string may be basedon one or more of: a symbol embedding; a unified vector for symbols ofan alphabet that is different from the alphabet used in the string; avector for frequently occurring punctuation marks; a vector for rarepunctuation marks; or a vector for numerals.

For example, FIG. 4 depicts an example of generating a sequence ofvectors 400 to provide to a machine learning unit 320 to obtain aprobability of occurrence 331. The example depicts generating thesequence of vectors 400 for string 311 of plurality of strings 310depicted in FIG. 3. Each vector of the sequence of vectors 400 maycorrespond to each symbol in the string 311. For example, vector 411 maycorrespond to symbol 311 a of string 311 and vector 412 may correspondto symbol 311 b of string 311. Each vector (e.g., vector 411, vector412, etc.) of the sequence of vectors 400 may be derived by joiningtogether a first vector 440 and a second vector 410 for each symbol(e.g., symbol 311 a, symbol 311 b, etc.) of the string 311,respectively.

The first vector 440 may comprise the maximum length of word for a givensymbol in the string. For example, first vector 440 of vector 411 mayconsist of one element for the maximum value for the symbol 311 a. Themaximum length of a word for each symbol may correspond to length of alongest possible word within the string that starts with the symbol. Insome examples, the longest possible word is found in a dictionary. Forexample, for the symbol 311 a, the longest possible word within string311 found in a dictionary that starts with the symbol 311 a consists ofthe three symbols in range 415. Thus, for symbol 311 a, the length ofthe longest possible word within the string is three, which isrepresented by “l₁.” Accordingly, the value of the element of firstvector 440 corresponding to the first symbol 311 a of string 311 is“l₁.” Similarly, the longest possible word within string 311 startingwith the symbol 311 b spans for two symbols and is represented by “l₂.”Accordingly the maximum length of the word for the symbol 311 b isassigned as “l₂.” The maximum lengths (e.g., l₁-l₉) of each symbol maybe determined using the same technique and be entered into first vector440 corresponding to each of the symbols of the string 311.

In the example discussed above with regards to the strings received froma text recognition system, the plurality of strings consisted of thestrings 1) “steamboat,” 2) “s1eamboat,” 3) “sleamboat,” and 4)“sieamboat.” For the string “steamboat,” “steam” is a word and“steamboat” is another word within the string that starts with thesymbol “s.” The maximum length of a word for the symbol “s” is thelength of the longest possible word “steamboat” within the string thatis found in a dictionary. Thus, the length corresponding to symbol “s”is nine. In another example, for the symbol “t,” “tea” is a dictionaryword and “team” is another dictionary word, with “team” being thelongest possible word appearing within the string. Thus, the maximumlength corresponding to the symbol “t” is four. A vector containing thevalue of maximum length may be derived corresponding to each symbol ofthe string. If no symbol sequences starting with the symbol were foundin the dictionary, then the maximum length could be zero.

The second vector 410 for a given symbol may comprise a symbol vectorfor the given symbol. In an implementation, the symbol vector for thegiven symbol may be formed based on symbol embeddings. Using symbolembeddings (or word embeddings), a symbol or a word may be representedvia vectors. Symbol embeddings are used to represent symbols in analphabet by mapping the symbols to vectors of real numbers. Using symbolembeddings, an entire alphabet can be formalized into a look-up table.The lookup table may be a hash table with unique vector for every symbolof the alphabet. Additionally, the look-up table may contain or be basedon additional vectors other than the symbol vectors. In some examples,the additional vectors may include a single or unified vector forsymbols of a different alphabet than the alphabet of the string. Thelook-up table may also include a vector for all frequently occurringpunctuation marks, a vector for all rare punctuation marks, vector fornumerals, etc. The vectors in the look-up table may be previouslytrained embeddings. In order to generate the symbol vector for a givensymbol of a string, the look-up table may be used to identify thecorresponding values for the given symbol of the string. In someembodiments, the symbol vector may have a dimension of 24. That is, thesymbol vector for the given symbol may consist of 24 elements in thesymbol vector.

Each vector of the sequence of vectors 400 corresponding to each symbolmay be derived by concatenating the first vector 440 and second vector410 together into one vector 450. For example, vector 411 is generatedfor symbol 311 a of string 311 by joining together the first vector 440for symbol 311 a and second vector 410 for symbol 311 a. Each of thecombined vectors for each symbol is then placed together to form thesequence of vectors 400. In the example of FIG. 4, the sequence ofvectors 400 consists of nine vectors, one vector each for each symbol311 a-311 i of string 311. Each of the nine vectors is derived bycombining a first vector 440 of maximum length and a second vector 410of symbol vector for the corresponding symbol.

Referring back to FIG. 2, at block 230, the computer system may provideto a machine learning unit the first sequence of vectors for each stringof the plurality of strings. For example, FIG. 3 shows arrow 318entering as an input to machine learning unit 320. As shown in furtherdetails in FIG. 4, the first sequence of vectors 400 is provided to themachine learning unit 320 for the string 311. The machine learning unitmay include a first fully connected layer 422 and a second fullyconnected layer 424 to apply matrix transformation on the first sequenceof vectors 400 for string 311. A batch normalization function and arectifier linear unit (ReLU) activation function are applied on a firstoutput of the first fully connected layer and on a second output of thesecond fully connected layer. In an example, the dimension of the firstand/or the second layer may be 256. A third fully connected layer to beused as an output layer. A sigmoid activation function may be applied ona third output of the third fully connected layer.

Referring back to FIG. 2, at block 240, the computer system may obtainfrom the machine learning unit a probability of occurrence of eachstring of the plurality of strings. The probability of occurrence for agiven string having a value nearest to ‘one’ relative to the probabilityof occurrence values of the remaining strings indicates that the givenstring has the highest probability of occurrence. The probability valuecan be identified by real numbers, generally ranging from zero to 1, butthe values may consist of intermediate values such as 0.1, 0.2, or 0.9,etc. FIG. 3 depicts arrow 319 coming out of machine learning unit 320 asan output. The machine learning unit may provide as an output aprobability of occurrence of each string of the plurality of strings.The probability value may be expressed using a vector, where each vectorcomponent corresponds to one string. For example, a vector 350containing four probability values has been produced as an output ofmachine learning unit 320. The value of each component of the vector 350may represent the probability of occurrence value corresponding to eachof the strings 310. The value of the component of the vector that isclosest to the value ‘one’ compared to the values of the othercomponents corresponds to the string having the highest probability ofoccurrence. For example, the first component of vector 350 contains thehighest probability value 331 with a value of “1” amongst all othervalues of “0.” Thus, in an example if the plurality of strings reflectsdifferent possibilities of a word to be recognized, the string thatproduces the highest probability value may be the string that representsthe most likely possibility of word recognition. FIG. 4 depictsobtaining from the machine learning unit 320 a probability of occurrence331 for string 311. The probability 331 is provided for the targetsymbol 311 e, given the context prior to the target symbol and thecontext following the target symbol.

FIG. 5 depicts an example computer system 500 which can perform any oneor more of the methods described herein, in accordance with one or moreaspects of the present disclosure. In one example, computer system 500may correspond to a computing device capable of performing method 200 ofFIG. 2. The computer system 500 may be connected (e.g., networked) toother computer systems in a LAN, an intranet, an extranet, or theInternet. The computer system 500 may operate in the capacity of aserver in a client-server network environment. The computer system 500may be a personal computer (PC), a tablet computer, a set-top box (STB),a personal Digital Assistant (PDA), a mobile phone, a camera, a videocamera, or any device capable of executing a set of instructions(sequential or otherwise) that specify actions to be taken by thatdevice. Further, while only a single computer system is illustrated, theterm “computer” shall also be taken to include any collection ofcomputers that individually or jointly execute a set (or multiple sets)of instructions to perform any one or more of the methods discussedherein.

The exemplary computer system 500 includes a processing device 502, amain memory 504 (e.g., read-only memory (ROM), flash memory, dynamicrandom access memory (DRAM) such as synchronous DRAM (SDRAM)), a staticmemory 506 (e.g., flash memory, static random access memory (SRAM)), anda data storage device 518, which communicate with each other via a bus530.

Processing device 502 represents one or more general-purpose processingdevices such as a microprocessor, central processing unit, or the like.More particularly, the processing device 502 may be a complexinstruction set computing (CISC) microprocessor, reduced instruction setcomputing (RISC) microprocessor, very long instruction word (VLIW)microprocessor, or a processor implementing other instruction sets orprocessors implementing a combination of instruction sets. Theprocessing device 502 may also be one or more special-purpose processingdevices such as an application specific integrated circuit (ASIC), afield programmable gate array (FPGA), a digital signal processor (DSP),network processor, or the like. The processing device 502 is configuredto execute instructions for performing the operations and stepsdiscussed herein.

The computer system 500 may further include a network interface device508. The computer system 500 also may include a video display unit 510(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), analphanumeric input device 512 (e.g., a keyboard), a cursor controldevice 514 (e.g., a mouse), and a signal generation device 516 (e.g., aspeaker). In one illustrative example, the video display unit 510, thealphanumeric input device 512, and the cursor control device 514 may becombined into a single component or device (e.g., an LCD touch screen).

The data storage device 518 may include a computer-readable medium 528on which the instructions 522 embodying any one or more of themethodologies or functions described herein is stored. The instructions522 may also reside, completely or at least partially, within the mainmemory 504 and/or within the processing device 502 during executionthereof by the computer system 500, the main memory 504 and theprocessing device 502 also constituting computer-readable media. Theinstructions 522 may further be transmitted or received over a networkvia the network interface device 508.

While the computer-readable storage medium 528 is shown in theillustrative examples to be a single medium, the term “computer-readablestorage medium” should be taken to include a single medium or multiplemedia (e.g., a centralized or distributed database, and/or associatedcaches and servers) that store the one or more sets of instructions. Theterm “computer-readable storage medium” shall also be taken to includeany medium that is capable of storing, encoding or carrying a set ofinstructions for execution by the machine and that cause the machine toperform any one or more of the methodologies of the present disclosure.The term “computer-readable storage medium” shall accordingly be takento include, but not be limited to, solid-state memories, optical media,and magnetic media.

Although the operations of the methods herein are shown and described ina particular order, the order of the operations of each method may bealtered so that certain operations may be performed in an inverse orderor so that certain operation may be performed, at least in part,concurrently with other operations. In certain implementations,instructions or sub-operations of distinct operations may be in anintermittent and/or alternating manner.

It is to be understood that the above description is intended to beillustrative, and not restrictive. Many other implementations will beapparent to those of skill in the art upon reading and understanding theabove description. The scope of the disclosure should, therefore, bedetermined with reference to the appended claims, along with the fullscope of equivalents to which such claims are entitled.

In the above description, numerous details are set forth. It will beapparent, however, to one skilled in the art, that the aspects of thepresent disclosure may be practiced without these specific details. Insome instances, well-known structures and devices are shown in blockdiagram form, rather than in detail, in order to avoid obscuring thepresent disclosure.

Some portions of the detailed descriptions above are presented in termsof algorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations are the means used by those skilled in the dataprocessing arts to most effectively convey the substance of their workto others skilled in the art. An algorithm is here, and generally,conceived to be a self-consistent sequence of steps leading to a desiredresult. The steps are those requiring physical manipulations of physicalquantities. Usually, though not necessarily, these quantities take theform of electrical or magnetic signals capable of being stored,transferred, combined, compared, and otherwise manipulated. It hasproven convenient at times, principally for reasons of common usage, torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise, as apparent from the followingdiscussion, it is appreciated that throughout the description,discussions utilizing terms such as “receiving,” “determining,”“selecting,” “storing,” “setting,” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

The present disclosure also relates to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, or it may comprise a general purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but not limited to, any type of diskincluding floppy disks, optical disks, CD-ROMs, and magnetic-opticaldisks, read-only memories (ROMs), random access memories (RAMs), EPROMs,EEPROMs, magnetic or optical cards, or any type of media suitable forstoring electronic instructions, each coupled to a computer system bus.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems will appear as set forth in thedescription. In addition, aspects of the present disclosure are notdescribed with reference to any particular programming language. It willbe appreciated that a variety of programming languages may be used toimplement the teachings of the present disclosure as described herein.

Aspects of the present disclosure may be provided as a computer programproduct, or software, that may include a machine-readable medium havingstored thereon instructions, which may be used to program a computersystem (or other electronic devices) to perform a process according tothe present disclosure. A machine-readable medium includes any procedurefor storing or transmitting information in a form readable by a machine(e.g., a computer). For example, a machine-readable (e.g.,computer-readable) medium includes a machine (e.g., a computer) readablestorage medium (e.g., read only memory (“ROM”), random access memory(“RAM”), magnetic disk storage media, optical storage media, flashmemory devices, etc.).

The words “example” or “exemplary” are used herein to mean serving as anexample, instance, or illustration. Any aspect or design describedherein as “example” or “exemplary” is not necessarily to be construed aspreferred or advantageous over other aspects or designs. Rather, use ofthe words “example” or “exemplary” is intended to present concepts in aconcrete fashion. As used in this application, the term “or” is intendedto mean an inclusive “or” rather than an exclusive “or”. That is, unlessspecified otherwise, or clear from context, “X includes A or B” isintended to mean any of the natural inclusive permutations. That is, ifX includes A; X includes B; or X includes both A and B, then “X includesA or B” is satisfied under any of the foregoing instances. In addition,the articles “a” and “an” as used in this application and the appendedclaims should generally be construed to mean “one or more” unlessspecified otherwise or clear from context to be directed to a singularform. Moreover, use of the term “an embodiment” or “one embodiment” or“an implementation” or “one implementation” throughout is not intendedto mean the same embodiment or implementation unless described as such.Furthermore, the terms “first,” “second,” “third,” “fourth,” etc. asused herein are meant as labels to distinguish among different elementsand may not necessarily have an ordinal meaning according to theirnumerical designation.

What is claimed is:
 1. A method comprising: generating, by a processingdevice, a sequence of vectors based at least on a maximum length of wordfor each symbol of a plurality of symbols in a string, wherein themaximum length corresponds to length of a longest possible word withinthe string that starts with the symbol, the longest possible wordcomprising one or more of a word: 1) with a commonplace meaning, or 2)found in a dictionary; providing the sequence of vectors for the stringto a machine learning unit; and obtaining a probability of occurrence ofthe string from the machine learning unit.
 2. The method of claim 1,wherein each vector of the sequence of vectors corresponds to a symbolof the plurality of symbols in the string.
 3. The method of claim 2,wherein each vector of the sequence of vectors is derived by joiningtogether a first vector comprising the maximum length of word for agiven symbol in the string and a second vector comprising a symbolvector for the given symbol in the string.
 4. The method of claim 3,wherein the symbol vector for each symbol in the string is based on oneor more of: a symbol embedding; a unified vector for symbols of analphabet that is different from the alphabet used in the string; avector for frequently occurring punctuation marks; a vector for rarepunctuation marks; or a vector for numerals.
 5. The method of claim 1,further comprising: receiving a plurality of strings comprising thestring, wherein each string of the plurality of strings comprises aplurality of symbols, and wherein generating a sequence of vectors isperformed for each string of the plurality of strings.
 6. The method ofclaim 5, wherein each string of the plurality of strings differs fromeach other string of the plurality of strings by one symbol, the onesymbol being located on a same position of each string.
 7. The method ofclaim 5, wherein the probability of occurrence for a given string havinga value nearest to ‘one’ relative to probability of occurrence values ofthe remaining strings indicates that the given string has the highestprobability of occurrence.
 8. The method of claim 1, wherein the machinelearning unit comprises: a first fully connected layer and a secondfully connected layer to apply matrix transformation on the sequence ofvectors for each string; and a third fully connected layer to be used asan output layer.
 9. The method of claim 8, wherein a batch normalizationfunction and a rectifier linear unit activation function are applied ona first output of the first fully connected layer and on a second outputof the second fully connected layer, and wherein a sigmoid activationfunction is applied on a third output of the third fully connectedlayer.
 10. A system comprising: a memory device storing instructions;and a processing device coupled to the memory device, the processingdevice to execute the instructions to: generate a sequence of vectorsbased at least on a maximum length of word for each symbol of aplurality of symbols in a string, wherein the maximum length correspondsto length of a longest possible word within the string that starts withthe symbol, the longest possible word comprising one or more of aword: 1) with a commonplace meaning, or 2) found in a dictionary;provide the sequence of vectors for the string to a machine learningunit; and obtain a probability of occurrence of the string from themachine learning unit.
 11. The system of claim 10, wherein each vectorof the sequence of vectors corresponds to a symbol of the plurality ofsymbols in the string.
 12. The system of claim 11, wherein each vectorof the sequence of vectors is derived by joining together a first vectorcomprising the maximum length of word for a given symbol in the stringand a second vector comprising a symbol vector for the given symbol inthe string.
 13. The system of claim 12, wherein the symbol vector foreach symbol in the string is based on one or more of: a symbolembedding; a unified vector for symbols of an alphabet that is differentfrom the alphabet used in the string; a vector for frequently occurringpunctuation marks; a vector for rare punctuation marks; or a vector fornumerals.
 14. The system of claim 10, wherein the processing device isfurther to: receive a plurality of strings comprising the string,wherein each string of the plurality of strings comprises a plurality ofsymbols, and wherein generating a sequence of vectors is performed foreach string of the plurality of strings.
 15. The system of claim 14,wherein each string of the plurality of strings differs from each otherstring of the plurality of strings by one symbol, the one symbol beinglocated on a same position of each string.
 16. The system of claim 14,wherein the probability of occurrence for a given string having a valuenearest to ‘one’ relative to probability of occurrence values of theremaining strings indicates that the given string has the highestprobability of occurrence.
 17. The system of claim 10, wherein themachine learning unit comprises: a first fully connected layer and asecond fully connected layer to apply matrix transformation on thesequence of vectors for each string; and a third fully connected layerto be used as an output layer.
 18. The system of claim 17, wherein abatch normalization function and a rectifier linear unit activationfunction are applied on a first output of the first fully connectedlayer and on a second output of the second fully connected layer, andwherein a sigmoid activation function is applied on a third output ofthe third fully connected layer.
 19. A non-transitory computer-readablestorage medium storing instructions that, when executed by a processingdevice, cause the processing device to: generate a sequence of vectorsbased at least on a maximum length of word for each symbol of aplurality of symbols in a string, wherein the maximum length correspondsto length of a longest possible word within the string that starts withthe symbol, the longest possible word comprising one or more of aword: 1) with a commonplace meaning, or 2) found in a dictionary;provide the sequence of vectors for the string to a machine learningunit; and obtain a probability of occurrence of the string from themachine learning unit.
 20. The non-transitory computer-readable storagemedium of claim 19, wherein the machine learning unit comprises: a firstfully connected layer and a second fully connected layer to apply matrixtransformation on the sequence of vectors for each string; and a thirdfully connected layer to be used as an output layer.